Closes #3006

Description Of Changes

For consent reporting, and for other purposes like building gpp strings, we need records of whether consent was served, even if the user chose to not save any preferences.

We don’t necessarily record each time an API request is made to get consent, but instead whenever the components actually show something to the user. Therefore, add a new endpoint for the frontend to call when they serve consent via a particular component. Add new tables to store that consent was served. Update the requests that save preferences to optionally pass in the id where we served the preference to be able to calculate conversion in the future. When retrieving preferences for a given user embedded on notices, also return whether that notice was served.

Code Changes

Some new tables were added and several methods abstracted to share code between when saving preferences and recording that consent is served, as we track very similar details

• Add new servednoticehistory to track every single time a version of a notice is served for consent reporting
• Add new lastservednotice table to consolidate the last time a notice was served, across any version, for all the known identifies of a user
• Add new column: privacypreferencehistory.served_notice_history_id to link the record between a notice being served, and the preferences being saved with respect to that notice
• When getting experiences via the API with a fides user device id query param, add whether the current notice was served or an outdated version of the notice was served to the embedded notice response
• Create a ConsentReportingMixin to share common fields between PrivacyPreferenceHistory and ServedNoticeHistory tables
• Create upsert_last_saved_record to share common code between upserting a CurrentPrivacyPreference or a LastServedNotice record
• Add new endpoints to save when consent was served, one for the privacy center, and one for banner/overlay: PATCH "/consent-request/{consent_request_id}/notices-served" and PATCH "/notices-served"
• Generalized verify_privacy_notice_and_historical_records to share with multiple API endpoints
• Generalized supplement_request_with_user_and_experience_details to share when saving preferences and the fact that notices were served. Both locations record a lot of information about the user.
• Dry up classify_identities_for_privacy_center_consent_reporting to share with multiple locations

Steps to Confirm

• nox -s dev -- shell
• Get a valid experience id and notice history id from this response.
  • GET localhost:8080/api/v1/privacy-experience?region=us_ca
• PATCH that the specific notice has been served. This is something the frontend will call when they serve notices in a component, by patching all the notice histories here in a list. If it's a banner, and no notices are shown, send all the notices that are in play here.
  PATCH http://localhost:8080/api/v1/notices-served

{
  "browser_identity": {
    "fides_user_device_id": "3e2e5f7c-5367-47d4-b672-9427626c71db"
  },
  "privacy_notice_history_ids": [
    "pri_52008ca3-406e-420f-8509-9429c73ef859"
  ],
  "privacy_experience_id": "pri_6ec7166a-28a8-49d9-9984-9b429d393d11",
  "user_geography": "us_ca",
  "acknowledge_mode": false,
  "serving_component": "overlay"
}

• Take note of the served_notice_history_id(s) in the response
• Update user consent preferences, linking to the served notice in the request body, so we can link conversion.
  PATCH http://localhost:8080/api/v1/privacy-preferences

{
  "browser_identity": {
    "fides_user_device_id": "3e2e5f7c-5367-47d4-b672-9427626c71db"
  },
  "preferences": [
    {
      "privacy_notice_history_id": "pri_52008ca3-406e-420f-8509-9429c73ef859",
      "preference": "opt_in",
      "served_notice_history_id": "ser_e83dd23c-cc82-44bc-9046-824089f80e2e"
    }
  ],
  "policy_key": "default_consent_policy",
  "privacy_experience_id": "pri_6ec7166a-28a8-49d9-9984-9b429d393d11",
  "user_geography": "us_ca",
  "method": "button"
}

• Get Experiences again, but with your fides user device id as a query param, so you can both see your saved preference and the fact that the notice was served embedded on the notice in the response:
  GET http://localhost:8080/api/v1/privacy-experience?show_disabled=true&region=us_ca&fides_user_device_id=3e2e5f7c-5367-47d4-b672-9427626c71db
  Truncated Response:

{
  "items": [
    {
      "region": "us_ca",
      "component": "overlay",
      "experience_config": {},
      "id": "pri_6ec7166a-28a8-49d9-9984-9b429d393d11",
      "created_at": "2023-07-06T13:28:35.171060+00:00",
      "updated_at": "2023-07-06T13:28:35.171060+00:00",
      "show_banner": true,
      "privacy_notices": [
        {
          "regions": [
            "us_ca"
          ],,
          "privacy_notice_history_id": "pri_52008ca3-406e-420f-8509-9429c73ef859",
          "default_preference": "opt_out",
          "current_preference": "opt_in",
          "outdated_preference": null,
          "current_served": true,
          "outdated_served": null
        }
      ]
    }
  ],
  "total": 1,
  "page": 1,
  "size": 50
}

• Update the original privacy notice - changing the name is sufficient
  PATCH http://localhost:8080/api/v1/privacy-notice

[
  {
    "name": "new_name",
    "id": "pri_8673c052-419a-4862-acee-410a8ade45a5"
  }
]

• Now that when you GET experiences, the notice returns that an outdated version of the notice was served to the user, and that we have an outdated preference saved as well

GET http://localhost:8080/api/v1/privacy-experience?show_disabled=true&region=us_ca&fides_user_device_id=3e2e5f7c-5367-47d4-b672-9427626c71db
Truncated Response:

{
  "items": [
    {
      "region": "us_ca",
      "component": "overlay",
      "experience_config": {},
      "id": "pri_6ec7166a-28a8-49d9-9984-9b429d393d11",
      "created_at": "2023-07-06T13:28:35.171060+00:00",
      "updated_at": "2023-07-06T13:28:35.171060+00:00",
      "show_banner": true,
      "privacy_notices": [
        {
          "regions": [
            "us_ca"
          ],,
          "privacy_notice_history_id": "pri_52008ca3-406e-420f-8509-9429c73ef859",
          "default_preference": "opt_out",
          "current_preference": null,
          "outdated_preference": "opt_in",
          "current_served": null,
          "outdated_served": true
        }
      ]
    }
  ],
  "total": 1,
  "page": 1,
  "size": 50
}

• There are separate endpoints for the privacy center, as these endpoints also let you record consent against verified identities like email and phone number. Create a consent request and note the returned consent_request_id.
  POST http://localhost:8080/api/v1/consent-request

{
  "email": "[email protected]",
  "fides_user_device_id": "3e2e5f7c-5367-47d4-b672-9427626c71db"
}

• PATCH that notices were served in the privacy center. Note the returned served_notice_history_id
  PATCH http://localhost:8080/api/v1/consent-request/<consent_request_id>/notices-served

{
  "browser_identity": {
    "fides_user_device_id": "3e2e5f7c-5367-47d4-b672-9427626c71db"
  },
  "privacy_notice_history_ids": [
    "pri_52008ca3-406e-420f-8509-9429c73ef859"
  ],
  "privacy_experience_id": "pri_6ec7166a-28a8-49d9-9984-9b429d393d11",
  "user_geography": "us_ca",
  "acknowledge_mode": false,
  "serving_component": "privacy_center"
}

• PATCH preferences in the privacy center with the served_notice_history_id in the request body
• PATCH http://localhost:8080/api/v1/consent-request/con_40577f7d-55ae-4067-bb1b-5d7cc795bfcf/privacy-preferences

{
  "browser_identity": {
    "fides_user_device_id": "3e2e5f7c-5367-47d4-b672-9427626c71db"
  },
  "preferences": [
    {
      "privacy_notice_history_id": "pri_52008ca3-406e-420f-8509-9429c73ef859",
      "preference": "opt_in",
      "served_notice_history_id": "ser_64565af7-0686-4e69-9100-352d1d11dfce"
    }
  ],
  "policy_key": "default_consent_policy",
  "privacy_experience_id": "pri_6ec7166a-28a8-49d9-9984-9b429d393d11",
  "user_geography": "us_ca",
  "method": "button"
}

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation:
  • documentation complete, PR opened in fidesdocs
  • documentation issue created in fidesdocs
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md
• For API changes, the Postman collection has been updated Postman is not working on my machine at all at the moment, I've spent too much time trying to get this up and running